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INTRODUCTION 


Educators, policymakers, philanthropists, and others have worked for decades to improve education by 
implementing new policies and practices to better serve students. But the desired improvement has been 
elusive - more sporadic (and anecdotal) than systemic. Success can be difficult to sustain, and what works 
in one school may not work on the campus down the street. 


Research-based initiatives should drive what happens in classrooms and schools. However, too often we 
see that new education initiatives fade away or have mixed results over time. Commonly, the “what" - the 
new program or initiative - gets most of the attention. But “how” that change will happen gets too little 
focus. 


Why? What is missing? 


Our recent research with four district partners revealed certain conditions that need to be in place for 
practice and policy change to occur. While our research is specific to Principal Talent Management (PTM), 
what we describe as the practices and policies for recruiting, supporting, and retaining effective principals, 
we believe these conditions may also apply to other efforts to improve schools for students and families. 


CONDITIONS FOR CHANGE 


1. Leadership Conditions 3. Cultural Conditions 


The superintendent is visibly committed to 
the project. 


The team has a sense of humility about 
the need to improve and values outside 


expertise. 


Resources (people and time) are allocated to 
the work. The team regularly shows an eagerness to 


problem solve. 
The school board is neutral or supportive 


regarding the project. 


2. Capacity Conditions 


A key deputy with decision-making authority 
leads the implementation. 


The implementation team is deliberately 
cross-functional in composition and 
includes the key stakeholder (principals in 
our project). 


The roles most central to the work and 
proposed change are identified and 
elevated. (In our project, this is the principal 
supervisors, who not only directly support 
principals but also serve as critical conduits 
between central offices and campuses.) 
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The team values continuous improvement 
over a checklist of accomplishments. 


4. Performance Conditions 


The team sets and uses meaningful goals to 
prioritize work. 


The team uses both data and calculated risk 
to inform strategy. They avoid getting stuck 
when strong data is not available. 


The team uses meeting time effectively to 
drive action. 


WHY PRINCIPAL TALENT MANAGEMENT? 


Recruiting, supporting, and retaining effective principals is key to improving schools because of the role 
strong principals play in school quality.'* They lead instruction, manage teachers, welcome parents and 
the community to campus, and handle the overall morale, culture, and operations of the school.24°°” 
Research links quality principals to improvements in areas like teacher retention and satisfaction, climate, 
student attendance, and student achievement.®”!°""? None of this is surprising. People like to work for 
great bosses who help everyone get better together. 


Why does PTM matter for students? Recent research highlights the importance of using PTM strategies to 
improve schools. Six districts that improved or implemented comprehensive PTM strategies through The 
Wallace Foundation’s Principal Pipeline Initiative created positive outcomes for principals, schools, and 
students.'? For example, principals placed after the new talent management systems were established 
outperformed their peers in comparison schools. Also, they were more likely to remain in their jobs longer. 
And students in those principals’ schools performed better than students in comparison schools. 


THE SCHOOL LEADERSHIP INITIATIVE 


To test our PTM Framework, we developed the School Leadership Initiative (SLI). We created a cohort of 
four school districts from across the country and focused on helping them improve their PTM practices 
and policies. The SLI has two goals: 1) Support the cohort districts in creating a stronger pool of effective 
principals who will stay in their schools longer, and 2) Learn with our cohort and then share our findings 
with districts and educators across the country. 


Our theory of change is anchored in this notion: If districts improve how they develop, support, and retain 
principals, students will ultimately benefit. 


Figure 1. District Cohort Theory of Change 


YEARS 1-3 YEARS 2-3 YEARS 3-5 YEARS 5+ 


Principals improve 
campus outcomes 
like culture and 
climate as well as 
teacher support and 
retention. 


When a district has 


Great teachers, 
supported by 
principals and central 
office, improve 
student outcomes. 


Principals are more 
satisfied in their 


effective policies and 

practices to recruit, select, — | roles and feel more 

support, and retain great embedded in their 
organization. 


: 


Principals feel 
more connected 
to central office 
staff for support. 


principals.... 
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Effective Implementation 


As in other complex human organizations, change in school districts can be difficult.'*"° We know 

this from research, and we know it from our own experience in the field. Change management and 
organizational behavior typically get substantive attention in business, yet we rarely support educators in 
the critical work of change (beyond offering some platitudes about teamwork and perhaps an inspirational 
poster in the hallway). We rarely hear talk around why new education initiatives face implementation 
challenges, yet we make enormous investments of resources (people, money, and time) in new ideas. 
This focus on implementation is even more important given the approximately $190 billion from the 
Elementary and Secondary School Emergency Relief (ESSER) Fund heading to states and districts to 
mitigate the impact of COVID-19. This a staggering investment, designed to be allocated quickly. The 
implementation stakes are higher than ever. 


We built the Effective Implementation (El) Framework to help address this implementation challenge in 
this research project. We included content, coaching, and tools related to the El Framework components 
in our program design along with PTM. Our work with the districts included equal measures of PTM and El 
content and expertise throughout. 


The Cohort Districts 


The four participating districts are located around the country (two in the Southwest, one in the East, 
and one in the West). The districts have between 60,000 and 90,000 students, and they each employ 60 
to 150 principals. In all the districts, at least 40% of the students qualify for free or reduced lunches, and 
least 10% are classified as English Language Learners (ELL). Table 1 shows the general demographics of 
each district. We gave the districts letter designations to protect anonymity. 


Table 1. Key District Demographics 


District Number of Free & English Minority Principal Current 
Students Reduced Language Average Superintendent 
Lunch Learners Tenure Tenure (years) 
(years) 

A 80,000 50% 30% 70% 10 <3 

B 60,000 40% 10% 50% 3 <3 

Cc 90,000 80% 30% 90% 7 >3 

D 70,000 60% 30% 50% 5 >3 


Note: The demographic data are rounded to protect district confidentiality. For purposes of this report, a minority student is 
one who is either African American/Black, Hispanic American, Asian American, or Native American. 


Initial selection criteria: 


Superintendent and board support for the project. 
A willingness to commit a team of 10-15 district leaders and principals to travel to meetings and 
work on this project during school hours throughout the year. 

- A“readiness” to do the work, meaning that the district hadn't yet begun in-depth PTM work, but 
had started some work in at least one of PTM Framework's component areas (i.e. a compensation 
study and updated leadership framework). 
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No obvious signs of major dysfunction that could hinder the work, such as a school board in 
turmoil, a newsworthy scandal, high turnover rates, or the like. 
Districts that were urban or suburban with at least 50,000 students. 


We invited 70 districts, based on recommendations from experts around the country, to apply to the 
cohort. The program team conducted site visits to six finalists. Four districts were ultimately invited 
to join the cohort. The George W. Bush Institute is not a grant-making foundation, so no grant money 
was provided to the participating districts. All district travel expenses, materials, and dedicated expert 
resources were paid for by the Bush Institute. The districts’ investment was time. 


Selected districts commitments to participate: 


Send a team of 10-15 team members to three convenings per year for three years. 

Set implementation goals around PTM and work with a technical advisor to make progress toward 
those goals between convenings. 

Participate in research during the project and for at least two years afterward. 

Sign data-sharing agreements that allow the research team to collect data each year. 


The Intervention 


The program design is anchored in three key elements: 


Nine convenings of all four district teams (three per year for three years) that delivered PTM and El 
content, provided facilitated worktime, and featured guest experts along with the program team. 
See Appendix A for asample agenda. 

Dedicated (and experienced) district advisors serving as a coaches and advisors for each team. 

An annual diagnostic process to measure progress on both the PTM and El Frameworks (codified 
in a detailed lengthy report for each district built from its own data). See Appendix B for a redacted 
sample. 


The Diagnostic 


Each fall, an evaluation team collected data using surveys, interviews, and focus groups 

of principals, assistant principals, superintendents, and some central office staff. The 
interviewers used a structured guide aligned to a closed-coding framework and tailored 

to each interviewee's job role. The team also collected related artifacts. Data were coded 
according to the PTM and El Frameworks. This was a largely qualitative process; the survey 
scores were viewed descriptively (e.g., looking at means and comparing them to previous 
years as well as across districts). 


A team of raters were trained and normed on a five-point rubric that aligned to the 
frameworks, where a score of zero indicated that the district had no or poor practice ina 
component area and a score of four meant that a district received a rating of best practice. 
At least three raters scored all components for each district, and scores were normed in 

a norming call. The districts received a detailed report that included their scores with a 
narrative featuring salient evidence and recommendations about how the district could 
improve its scores over the next year. 


Figure 2 details the scoring rubric for the diagnostic process. 
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Figure 2. Diagnostic Scoring Rubric 


4 - Best Practice 3 - Strong 0-Poor/No 
Practice Practice 


(Models consistent 
practices with little (Effective practices (Poor practice with 
room for that are becoming little to no action 


improvement) more consistent) and/or lack of 
awareness of 
problems by 
top-level district 
U¥-To 3a 4]/0)] 


We scored the districts at the component level and the indicator level. We used those scores 
as qualitative data to show the progress each district made annually. This allowed us to see 
variation in implementation over time. 


District Progress 


Not surprisingly, there was variation between the districts, which allowed us to compare district practices 
to identify conditions that enhanced implementation. While all four districts made progress throughout 
the four years in the program, Districts C and D made more progress than Districts A and B. 


District A had mostly “emerging practice” ratings at the end of Year 3, and District B had only one 
component area in “strong practice.” Districts A and B had more scores that varied year over year, 
sometimes progressing one year only to fall behind again in the next. Districts C and D mostly made 
steady positive progress in PTM over time. 


Figure 3. PTM Progress 


Year 0 Year 1 Year 2 Year 3 
(Fall 2017) (Fall 2018) (Fall 2019) (Fall 2020) 


AB C DJA B.C DIA B.C DIA. B\C. OD 
Component ont 
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. Professional Learning eeeeeceeooe @ eleieee 


. Performance Evaluation eeeoeeeeeoeo oo e\e\eee 
. Compensation and Incentives |@ @ © |e ® @ |e @ o —(e ® e ® 
. Principal Supervisors | *Not measured |e 8 @ @|@ eo. e\e eee 
. Working Environment \eleejeieeeelee ee eeoee 
- Coherence 0|0'0|0| 0 00/0/00 00/00 Oe 


®@ Best @ Strong @ Emerging Poor Practice Poor / No 
Practice Practice Practice with Action Practice 
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We hypothesized that district teams that made more progress on PTM would also have higher scores on 
El components taught in the program. This proved true. District D had strong practice in almost every 
component area of El, while District C had strong practice in three of the five areas. Neither District A nor 
B had “strong practice” in any El component area, as Figure 4 shows. 


Figure 4: El Progress 


Year 2 Year 3 
(Fall 2019) (Fall 2020) 


Dene B Cc B Cc 


1. Initiate Change 
2. Engage Stakeholders 
3. Build Capacity 


4. Set Goals and Create Plan 


5. Execute, Reflect, and Improve 


*The El Framework underwent a substantial revision and validation process with experts between Years 1 and 2. 


Best ® Strong © Emerging Poor Practice Poor / No 
Practice Practice Practice with Action Practice 


To systematically examine the conditions that enhanced effective implementation, we compared the 
practices in Districts A and B (the “emerging districts”) to practices in Districts C and D (the “strong 
districts"). We reviewed data from the diagnostic and relied on our own observations. We used a 
qualitative approach by creating a coding scheme aligned to our frameworks, coding the diagnostic and 
observational data and grouping findings into themes. 
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THE CONDITIONS FOR CHANGE 


We found four sets of conditions that enhanced or supported implementation in the districts. 


1. Leadership Conditions 


Strong districts exhibited three leadership conditions: visible, committed leadership; resources allocated 
to the work; and school board support. 


Visible, committed leadership 


In strong districts, the superintendents did the following: 


Spoke publicly about the work in ways that showed how it tied to the districts’ overarching goal. 

-  Signaled to their leadership teams and others in the district that the work was a priority and that 
those doing it had full support from the superintendent's office. 

* Strategically attended working meetings or program sessions to be available for feedback from the 
team without micromanaging the work. 
Remained informed about the work. 


The teams took these actions as permission to prioritize the work. According to one interviewee in District 
D, “Once we learned and understood that our superintendent supported the work we were trying to do, 
we felt like we could dive in with both feet. | felt supported.” One interviewee in District C said, “It's coming 
directly from our chief who reports to our superintendent. So it comes from them just sending out the 
message that this is what we are here for. This is our role. And then, even in principal meetings, in various 
principal communications, just re-emphasizing that again. That it’s our job to be here to support you and 
help make things easier for you and you as a principal.” 


Superintendents in emerging districts were not as consistent in these practices. Neither spent time 
publicly tying the PTM work to the district goals, and they were not as up to date on the work as strong 
district superintendents. This resulted in a lack of vision. In District A, one interviewee said, "We need to 
develop a focus because it is just kind of all out there.” And, similarly, a District B interviewee said, “| really 
don't feel like | have a solid answer for what the exact vision is for the district.” 


Resources allocated 


Strong districts consistently allocated the following resources of time, people, and money to this work: 


* Funding to support additional positions, such as much-needed new principal supervisors; 
technology or software when needed, such as new evaluation software; and training of key 
personnel to build capacity. 
A willingness to move tasks and assignments from PTM team members’ portfolios so that they 
had time to work on implementation. 

- — In-district coverage so that the PTM team could travel to (or participate virtually in) the program's 
convenings. 


In strong districts, superintendents were willing to take a deep look at budgets to determine how to fund 
new positions or resources needed for this work. This meant making difficult choices to eliminate other 
budget items. One district relied on a grant to fund some of the positions specific to this work, and, when 
the grant ended, the superintendent found money to continue those key positions. Another worked with 
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community foundations that were willing to provide extra funding to support positions in the district long 
term. 


Emerging districts also committed resources to the work, but the commitment was inconsistent. For 
example, one district hired additional staff, but it was not enough to cover the work. And, to find the 
funding for those additional positions, contractors had to be cut. Unfortunately, the work of those 
contractors fell to the already overworked team, so the net result was no additional capacity. Also, the 
PTM teams in the emerging districts had the most difficult time attending convenings. In one district, only 
a limited number of team members could travel for some of the convenings. In another, the team could 
travel but was consistently interrupted by urgent calls from the superintendent. These districts did invest 
in various trainings to build capacity, but they were generally one-time, one-off investments that did not 
translate into practice changes in the districts. 


Board support 


In strong districts, the board was a supportive bystander of the work: 


- Aware of the work from the onset and regularly briefed on progress, updates, and decisions. 

-  Visibly neutral or supportive of the efforts and changes. 
Enabled the superintendent and leadership to make decisions, lead the implementation, and was 
open to recommendations. 


Contrary to our initial hypothesis, the board did not play an active role in the strong or emerging districts’ 
implementation. However, in strong districts, the superintendent appropriately updated the board, and, 
in turn, the board trusted the superintendent and their team to make the right decisions at the right time. 
One interviewee in District D said, “The superintendent does most of the board of education interfacing 
and education and collaborative efforts, and! agree with that. | think that's the way it should be. There is 
trust there.” 


2. Capacity Conditions 


Our interview data showed there is always more work to be done in a district than talent available to get 
it done, but the stronger districts more effectively used the talent they had to drive implementation. The 
following capacity conditions enhanced their implementation: having a key deputy to lead and oversee 
the work, understanding and effectively utilizing the important role of the principal supervisor, and having 
broad representation of district stakeholders involved as the work was implemented. 


Key deputy to lead 


In the strong districts, the first thing the superintendent did after being accepted into the cohort was to 
appoint a very high-level deputy to oversee the project. This deputy was responsible for the following: 


Initiating the work and setting the PTM vision (with the superintendent). 
Establishing the PTM team and ensuring the right people were engaged. 
- Being aliaison between internal teams, external contacts, and leadership. 
* Owning the PTM work and progress as a core responsibility of the deputy’s district role. 


This smoothed over implementation for a variety of reasons. First, it signaled to other staff in the district 


that this work was a priority of the superintendent's. Second, it helped speed the decision-making process 
for the PTM team. The deputy was able to make many decisions on the spot and had a direct line to the 
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superintendent for any decision that needed their feedback. Also, the deputy was able to step in and 
provide influence and cover when departments or high-level staff members were resistant to the initiative. 
In both strong districts, the deputy was savvy to other work happening in the district; as a result, the 
deputy could help the team prioritize and integrate the PTM implementation work into other areas. One 
interviewee in District D said, “The advantage is (the deputy’s) knowledge, (their) work with other groups, 
and (their) work with the superintendent. (They) are able to scan for what is going to be a problem for us. 
(They) know better who is going to agree with what or not. I'd say this work has been effective because of 
(the deputy’s) role in it.” 


This step was not taken in emerging districts. In District A, for example, a more junior person was 
responsible for the work. This staffer needed to manage up to more senior colleagues, which became 
a consistent struggle. This person also had no way to directly report to the superintendent or senior 
leadership. As a result, the work was often deprioritized, leaving team uncertain how to proceed and 
unclear about what members were and were not allowed to do. 


The role of the principal supervisor 


Strong districts recognized early the important role of the principal supervisor. They did this in the 
following ways: 


Giving principal supervisors autonomy to lead the PTM work. 
Providing principal supervisors with time and space to dedicate to PTM work. 

* Elevating and refining the role. 

- — Identifying opportunities to train and equip principal supervisors. 

- — Allowing the principal supervisors to serve as connectors between the implementation, the 
central office, and principals. 


Teams that boldly embraced and redefined the principal supervisor role accelerated their implementation 
and moved closer to their vision. Previous research shows that implementation is aided by linking 
resources at the central office to the school sites.'°'” Our study confirmed this and found that principal 
supervisors acted as the “glue” between school sites and the central office and between departments 
within the central office. 


Interestingly, principal supervisors were mentioned in over 90% of all interviews across all districts, which 
was more than any other role, including the superintendent. The role is unique because the principal 
supervisor holds authority within the district, navigates and mediates between the central office and 
school sites, and navigates among departments within the central office. According to one principal 
supervisor, “What we experience is a lack of continuity from different departments that support the 
campus principal. What | have personally seen is that when you sit in this chair, there's a different view 

in how we run a school than what some of the departments see. There's a disconnect from the various 
departments to campus. There is a reality check on various policies and/or practices.” Strong districts used 
the role of the principal supervisor to provide that “reality check" and to make sure that all understood the 
needs of the school sites. That helped smooth implementation of PTM policies. 


In elevating this position, one district even made principal supervisors reapply for their jobs after 
redefining the role. This was noted positively in the interview data. One principal from District C said, 
feel that the role has been developed intentionally, and it’s really made a big impact on me. They area 
thought partner, a coach. | feel like it's so much more productive than it used to be.” Also, throughout 
the principal supervisor interviews, there were no negative mentions of this process. Most described the 
reapplication as “rigorous.” One said, “I have faith that the people selected were really talented.” 


Hi 
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In addition to significant distractions in the emerging districts, principal supervisors did not feel that they 
had the authority or support to really take on the PTM work. When doubts existed about the work, and 
whether it would last, it was difficult for team members to put their full effort into it. A principal supervisor 
in District B explained, “Not knowing whether this (PTM work) is going to be given a thumbs up ora 
thumbs down can be frustrating and often blocks our efforts.” 


Broad representation, including principals 


Strong districts built implementation teams that had broad representation but cohesively worked 
together. They did this in the following ways: 


Including representation from multiple departments, such as human resources, leadership, 
academics, professional learning, and others. 
Leveraging principals, the beneficiary of the work, to prioritize their perspective and feedback and 
build buy-in. 

- Working together as a team (e.g., shared work plans, assigned roles and responsibilities, open 
communication, inclusive meetings, etc.) across departments and PTM workstreams. 


Strong districts used their PTM implementation teams effectively to increase productivity and take action. 
One key role on the team was that of principals. Since principals were the beneficiaries of the PTM work, 
they served as a sounding board and provided early feedback on ideas and plans. And they were able to 
help get other principals on board with ideas in some cases. 


Emerging districts also had some representation on their teams, and both emerging teams included 
principals. However, siloed work persisted. According to an interviewee in District A, “Because we have 

all these silos, | wouldn't even say that all district leadership even knows what's being rolled out.” And the 
team members struggled to understand their roles on the team, as well as how their role or department 
fit into the broader implementation picture. For example, one interviewee in District A said, “They don't 
know what part they need to be doing, and so if they don't know what they are supposed to be doing, 
they're also then not developing a system.” Work and assigned tasks frequently fell through the cracks and 
got pushed off. In District B, team members changed out frequently because of turnover, which slowed 
the work. 


3. Cultural Conditions 


Strong districts exhibited three cultural conditions: a sense of humility (coupled with an embrace of the 
district advisor), an eagerness to problem solve, anda valuation of continuous improvement. 


Humility 


In strong districts, evidence of humility was seen throughout the interview data. They did this in the 
following ways: 


Showing an awareness of the district's gaps and a willingness to learn from others (from the 
superintendent on down). 

* Seeking and accepting input from external experts and internal stakeholders from the central 
office to individual classrooms. 
Admitting mistakes and spotlighting successes on the team and others. 
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Assistance relationships, in which a novice is paired with an expert for the purposes of learning, 
can support districts’ culture and create “learning organizations,” research shows.'® In the PTM 


implementation, the district advisors personified 
the assistance relationship. The district advisors 
were critical thought partners for strong districts, 
which humbly took advantage of the advisors’ 
expertise to grow their own capacity. For example, 
the superintendent in District C said, “We all come 
to this with our biases, and | have mine. | want an 
internal and external partner to help check us and 
hold us accountable for bad decisions. We have 
more improvement to go.” In emerging districts, 
interviewees tended to list accomplishments with 
no awareness of gaps or areas of improvement. 
District advisors took more of a lead role in 
emerging districts, working to set up meetings and 
tracking actions to make sure they were completed. 
Strong districts, on the other hand, sought out the 
advice of the district advisors, but led the work 
themselves. 


Eagerness to problem solve 


Not only did the district advisors help build the 
content knowledge of the project team, but 
they also helped the districts stay focused on 
the work. One project team member described 
how they worked with their district advisor: 
“Normally, when [our district advisor] comes, 
every subcommittee sets a deadline on what 
we need to do.... And she brings us all together 
like, ‘Where are we with the work?” 


An interviewee from another district said, “This 
... has been transformational for our district. 
We have had critical conversations that were 
necessary. It was great for [our district advisor] 
to be there [to] help navigate, and it’s given us 
a focus that has been really good for us." 


Strong districts showed an urgency and eagerness to address PTM components and goals, while 
emerging districts struggled to get beyond identifying the problem. Strong districts did this in the 


following ways: 


- Having a clear understanding of the problem, a vision for what was possible, and clarity about why 


change was essential to supporting student outcomes. 
Focusing on measurable change over anecdotal or incidental success. 
Persisting through ambiguous and complex problems whose solutions have the potential for 


significant positive impact on students. 


Valuation of continuous improvement 


Strong districts had a culture of deep, embedded daily improvement-focused practice. They did this in the 


following ways: 


* Creating systems to reflect on data and progress to goals regularly and transparently. 
* — Replicating what is working and stopping what is not. 
» Adjusting along the way by seeking out and incorporating new strategies as needed (while 


maintaining a focus on feasibility). 


In strong districts, interviewees would ask their interviewers questions like, “Do you know anyone who 
does that well?” The strong district teams were always eager to learn more and get better at what they 
were implementing. As a matter of fact, the District C superintendent interrupted one interview to 

brainstorm with the interviewers, "So let's just talk that out for a minute if you have time... what if we...” 


Interviews from emerging districts read differently. While frustrations emerged, these were largely from 
principal and aspiring principal focus groups who were on the receiving end of many PTM policies and 
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practices. Interview transcripts of those on the PTM team, however, mostly listed accomplishments, 
particularly in Year 2 and Year 3. When any frustration was brought up, it was explained away and the 
conversation moved on. For example, one interview transcript reads as follows:'” 


District team member: “We updated our evaluation system, so now we have multiple 
measures, and we have put a lot of work into that. We are revising the language. 
Right now, it's kind of organic, and we're collecting data. For professional learning, 
we've worked hard on the model that the program taught us at the convening. We tell 
principals all about it. And we got a grant to help train our principal supervisors." 


Interviewer: “And | think last year you were going to set up principal supervisor 
training? How did that go?” 


District team member: “Well, we're kind of waiting to schedule. We just want to be sure 
we are clear on their roles and responsibilities and things like that. Our superintendent 
is really supportive, we're all just so busy trying to figure out what makes sense.... The 
other thing we are doing is that we're going to start holding informational sessions for 
assistant principals...” 


4. Performance Conditions 


The final theme that emerged was team performance during implementation. The diagnostic scores 
revealed that districts which had the most PTM growth also had the most growth on El. Strong districts 
exhibited three performance conditions: set and used meaningful goals to prioritize work, used data and 
calculated risk to inform strategy, and used meeting time effectively. 


Set and used meaningful goals to prioritize work 


Strong districts had teams that were able to set, articulate, and stick to goals, and they had confidence 
that they could achieve them. They did this in the following ways: 


- Established goals aligned to the vision and desired outcomes. 
- Set goals that are both urgent and feasible. 
Prioritized (and reprioritized) goals by considering impact and timing. 
* Communicated goals and progress, particularly quick wins, to all stakeholders. 
» Understood their current state of performance, including both gaps and areas of success as 
well as the root causes for the current state. 
- Used the current state assessment to guide planning and goal setting. 
Created high-quality implementation plans informed by their data that showcases an urgency 
and asense of possibility (but recognizes potential risks and includes mitigation plans). 


Used data and calculated risk to inform strategy 


Strong districts embraced their diagnostic data as an opportunity to learn and improve. At the same time, 
they were not surprised by their results as they were largely aware of their own areas of success and 
growth, given their ongoing feedback and reflection efforts. In District D, they surveyed each department 
to “make action steps of what to improve, and they celebrate the good feedback that was received.” 
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Used meeting time effectively 


Strong districts utilized meeting practices that promoted quality and efficiency. They did this in the 
following ways: 


- Meetings had clear objectives, reliable schedules, and critical participants. 

- Meetings involved facilitators, timekeepers, and/or other roles. 

» Meetings relied on team norms and utilized protocols and agendas. 

- Meetings resulted in a list of clear actionable next steps that included specific tasks, owners, and 
deadlines that were followed up on at the next meeting. 
Teams had structures in place to promote frequent collaboration including document 
management systems, subcommittees, shared project plans, etc. 


Strong districts had meetings that were productive, and they learned how to break the implementation 
team into subgroups to get more done. Their time together was respected and was used to make 
decisions and move work forward. One interviewee in District D described that their meetings do a “good 
job of providing enough time for discussion but also keeping us moving." 


Emerging districts struggled to use meeting time efficiently, often not following the agenda, erupting in 
side conversations, or disrespecting norms. One emerging district consistently overpacked its agendas, 
focusing on reporting instead of discussion, and often concluded meetings without accountable next 
steps. One interviewee in District A described the district's meetings as “not collaborative in a way that 
translates into action.” 


A Note on COVID-19 


It is important to note that COVID-19 began in Year 3 of the program. While this study was not meant 

to address the pandemic specifically, it was top of mind during the data analysis process, given the 
considerable impact on districts around the country. Early findings indicate that the pandemic may have 
emphasized district strengths and weaknesses that were present before it. For example, if the district 
had strong, visible leadership in prior years, that remained. Alternatively, if a district struggled to set and 
maintain goals, that was emphasized during the COVID-19 crisis. More work needs to be done to study 
the effects of the pandemic on implementation. 
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IMPLICATIONS FOR DISTRICTS (AND THEIR PARTNERS 
AND FUNDERS), POLICYMAKERS, AND RESEARCHERS 


We believe these four sets of conditions (leadership, capacity, cultural, and performance) are necessary 
when implementing a new idea, practice, policy, or initiative in school districts. And we believe that 
practitioners, funders, policymakers, and researchers should pay attention to the conditions when 
investing in or otherwise prioritizing resources for something new. Without a specific focus on 
implementation, we risk wasting valuable money and time, and, most critically, we will not deliver on our 
collective promise to help all students learn and succeed. 


Practice Implications 


For districts (and those that work with them frequently), these conditions can be used as a readiness 
assessment ahead of launching new work. In some cases, additional work and/or capacity building is 
needed before beginning large-scale implementation. Or it may be best to initiate a pilot or proof of 
concept effort first, with guardrails in place that align to the conditions. Districts that realize that they do 
not meet these conditions while in the middle of a large-scale implementation effort should take time to 
pause, reflect, and adjust. Districts, and their partners, should focus on building effective implementation 
skills in their district staff. 


Doesn't that delay important work? We are not suggesting that districts must plan and plan and plan 
before acting. We are saying that to ignore implementation will almost guarantee that the desired impact 
will not be achieved. That should not surprise us, and it is time for the field to acknowledge and address 
this reality. 


Table 2 offers a way to apply these conditions in a district. The left column describes each condition in 
more detail. The right proposes some critical questions that district leaders, partners, and funders could 
ask to determine if the district is ready to launch a new initiative. If these questions can be answered, 
the district is probably in a good place to implement change. If not, that likely indicates where some 
preimplementation work will provide the biggest payoff. 
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Table 2. Critical Questions for District Readiness 


Leadership 


Implementation is enhanced when the - Is the superintendent committed to the work, and 
superintendent is visibly committed to the work willing to be the public face of the work? 

and has the trust of a committed school board. The - How does the work tie to existing district goals? Is 
superintendent talks publicly about the work and the tie clear and strong? 

ties it to district goals. Resources are allocated to - Does the superintendent have a good working 

the work and the superintendent stays informed relationship with the board and have the autonomy 
about it. The superintendent has a good relationship to make critical resource allocation decisions? 

with the school board and keeps members updated » What communication channels does the 
appropriately. superintendent have to message the work 


internally and externally, and how frequently and 
effectively are they being used? 


Capacity 


Implementation is enhanced when districts use + Is there a deputy who has direct access to the 

their talents effectively to lead and support the work superintendent and authority to make decisions 
being implemented. A key deputy who has a direct available for this project? 

line to the superintendent and the authority to make + Does the implementation team include members 
decisions should be involved in the work. Principal from multiple departments in the district? Consider 
supervisors are critical to any implementation work departments that may affect or be affected by the 
that impacts the schools. Broad representation from work or departments that could become barriers 
multiple departments, including principals, should be to the work if not involved. 

on the implementation team. + Are arepresentative number of principals available 


to commit time to the work? Are there ways 
to support them in their buildings so they can 
dedicate attention to the work? 

+ Can principal supervisors commit to the work, 
and understand how their role could enhance 
implementation? 


Culture 
Implementation is enhanced when district cultures - Does the district have a culture of humility, and 
are defined by humility, innovation, and continuous are leaders and others willing to admit gaps or 
improvement. This is apparent when implementation mistakes in the spirit of improving? 
teams are aware of their gaps, willing to try new - Does the district focus on goals and 
approaches, and use data and external expertise to measurement? Is there a tolerance risk? 
continuously improve. - Is there a culture of learning in the districts? 
Performance 
Learning is enhanced when district teams take action + Does the district have a culture of using data to 
and stay focused. They have clarity and confidence learn about root causes and possible next steps? 
around the goals they set, but they are also willing + Does the district have a track record for setting 
and able to take calculated risks. They use data to goals and taking action? 
monitor progress and improve. * Does the district have a track record of utilizing 
effective implementation techniques? 
+ Is the district leadership team willing to take 


calculated risks and pilot new ideas? 
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Policy Implications 


How can policymakers support stronger implementation? Without requiring mandates, officials at the 
local, state, and federal level can assemble resources and convene experts to help districts as they 
grow their own capacity. In some cases, districts are reinventing the wheel each time they start to 
implement something new or get a new leader. State agencies and service centers can build their own 
implementation capacity as part of an effort to focus on improvement over compliance. 


Also, policymakers should consider the conditions of leadership, capacity, culture, and performance 
when creating mandates that require considerable implementation efforts. Directives to change - or 
new initiatives - that come down to districts and campuses without implementation support and/or 
direct connection to goals bring with them a big risk of failure. The new idea or intervention will likely 
be a struggle if implementation support is left to chance. The pain of a failing implementation may also 
turn educators off the new idea or program, even if it is of high quality. In other words, unsupported 
implementation can sink even the strongest ideas. 


Policymakers should determine which tools they can use to motivate and support districts to align with 
the conditions of leadership, capacity, cultural, and performance before implementation starts. That 

may include drafting rules around a preimplementation phase to begin before required implementation 
starts - and incentivizing the use of pilots or smaller-scale proof-of-concept implementations. This study 
reinforces the idea that we should expect variation in implementation, and policies should be designed to 
take that into account. Context matters, and districts could reasonably follow these conditions with some 
variation given their unique strengths and challenges. Measurement and evaluation strategies should be 
adjusted to take this into account as well, so that we better understand what works. 


Finally, policymakers should determine how and when district leaders build up their effective 
implementation skills. While some of these competencies are taught in graduate education programs, 
their applications are rarely robust or practical. How might credentialing and professional learning for 
administrators change if candidates used implementation case studies to build capacity of system 
leaders? How might leadership frameworks change if implementation skill was prioritized? 


Research Implications 


While the literature was helpful in framing this study, and literature on implementation in general has 
shifted over time to look at variation, more specificity would be helpful for practitioners. Theory is a 
helpful place to start - and it is critical in understanding what works, but we found that our districts 
needed theory plus practical concretes. That meant we sourced everything from principal supervisor job 
descriptions to school leadership talent pipeline maps to sample agendas for cross-functional teams to 
share with our teams in an informal “tool on demand" process. Team members wanted to talk through 
strategies on how to incorporate a recalcitrant colleague into the implementation effort. They wanted 

to practice how to succinctly update their superintendents or other senior leaders about both successes 
and challenges. Educators are clamoring for practical tools, and we, as a field, too often deliver theoretical 
jargon. 


This case study compared four districts after they had conducted implementation work for four years. It 
was too soon to use traditional quantitative outcomes to evaluate the work, but following this work over 
the long term would be beneficial. It would show us how implementation continues to vary over time, 
particularly as people transition in and out and new initiatives are introduced. 
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Finally, further research on the specific PTM policies and their ro 


le in the overarching talent management 


system would be helpful in guiding districts to focus their efforts. For example, does implementing 
practices and policies on recruitment and selection matter more than policies on compensation and 


incentives? 


Educating students is complex work and has very high stakes, gi 


to a young person's future. Doing well by students should guide 
are optimists, we are not naive enough to believe that that appli 
history play a critical role in implementation. We believe that ac 
conditions of leadership, capacity, culture, and performance wil 
focus on strong implementation, and the conditions that suppo 
status quo. It is a vote by the adults to do right by the students 


ven the importance of a quality education 
all adult decision-making. But while we 
es universally. A district's context and 
ulture missing one or more of the four 
struggle to improve. But districts that 

rt it, are actively resisting the pull of the 


in their charge. 
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APPENDIX A. SAMPLE CONVENING AGENDA 


ea 
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GEORGE W BUSH 
INSTITUTE 


Virtual School Leadership Initiative Convening 
October 28 — 30, 2020 
WEDNESDAY, OCTOBER 28 


12:00 P.M. — 1:00 P.M. CT OPTIONAL Session A 
Via Zoom 


1:30 P.M. — 4:00 P.M. CT Welcome / Expert Consultancy Session / Wrap Up 
Via Zoom 


THURSDAY, OCTOBER 29, 2020 


8:30 A.M. — 9:20 A.M. CT Session with President and Mrs. Bush 
Via Zoom 


9:30 A.M. — 12:00 P.M. CT Principal Talent Management (PTM) Session 
Via Zoom 


12:15 P.M.— 1:15 P.M. CT Superintendent Only Session / Lunch Break 
Via Zoom 


1:30 P.M. — 4:00 P.M. CT Effective Implementation (EI) Session 
Via Zoom 


FRIDAY, OCTOBER 30, 2020 


8:00 A.M. — 10:00 A.M. CT Facilitated Work Time: District A / District C 
Via Zoom 


10:00 A.M. — 10:30 A.M. CT Closing Session 
Via Zoom 


10:30 A.M. — 12:30 P.M. CT Facilitated Work Time: District B / District D 
Via Zoom 
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APPENDIX B. EXCERPT OF A DIAGNOSTIC REPORT 


PTM Component 4: Performance Evaluation 
Year 1 Score: 71 | Year 2 Score: 2 | Year 3 Score: 3 


Districts with model performance evaluation practices set clear and transparent standards and 
competencies against which school leadership performance is assessed. These standards are tightly and 
clearly aligned with research (see coherence indicators) as well as with other components of PTM. There 
is acommon understanding of what standards look like in practice, such that all principal supervisors and 
principals can easily understand and agree on what constitutes proficient practice for each competency 
(i.e., there is strong inter-rater reliability). Evaluations include multiple measures that are valid, reliable, 
and respected by principals. Districts with model evaluation practices have well-trained assessors (i.e., 
principal supervisors) who conduct their assessment via an iterative and transparent process. Their 
process is perceived by principals as appropriate and useful. Districts with model practices utilize the 
results of performance evaluations to drive the professional learning support they provide to principals. 


Model Practice 


Current Practice and Recommendations 


Based on diagnostic data collected over the last year, we assigned District D an overall rating of 3 (Strong 
Practice) in the area of principal performance evaluation. This is an active focus area for the district. The 
team has developed a School Leadership Framework and begun work to align the principal evaluation 
system to it. Principal supervisors have been more intentional about goal setting (tied to the evaluation) 
with principals and are referring to these goals during visits throughout the year. 


The district has also adopted a new evaluation tool to simplify the previous process that involved two 
tools. Measures for the evaluation include supervisor observations, artifacts, and a single measure 

of student achievement, but the team has worked to improve lines of evidence for each standard. 
Supervisors do some norming and calibration, and all supervisors had their principals do a self evaluation 
so that principals better understood the tool. Evaluation is not yet tightly tied to professional learning. 


To further strengthen performance evaluation in District D, we recommend that the district do the 
following: 


1. Continue the strong focus on the evaluation system, especially once crisis planning for the 
pandemic lessens. 

2. Establish a process for regular calibration among the principal supervisors. 

3. Consider additional measures to include in the principal performance evaluation, such as climate, 


student engagement, and student academic growth metrics that are consistent across principals. 


4. Similar to the recommendation in the “Professional Learning” section above, the district should 
work to closely tie performance evaluation data with the types of professional learning being 
offered (both group-based and individualized) to principals. 
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The chart below illustrates the district's current performance across key aspects of performance 
evaluation. 


Figure 5. Performance Evaluation Indicator Scores by Year 


4a1. Alignment 


4a2. Variety of measures 


4b1. Evaluation process 


4c1. Connections to professional learning 


Overall 


OYearO OYear1 @Year2 MyYear3 
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Below is an example of an indicator of PTM Component 4: Performance Evaluation. 


4b. Evaluation process 


4b1. Evaluation 
process 


4—- Best Practice 


3 -— Strong 
Practice 


Evaluation is regular, 
iterative, and 
transparent. Evaluation 
is conducted by 
supervisors who are well 
trained and normed. The 
cadence and results are 
reportedly appropriate 
and useful. 


Evaluation is 
regular, iterative, 
transparent, and 
conducted by 


supervisors who 
are trained and 
normed. 


Relevant Evidence: 


(Year 0 Score: 1) (Year 1 Score: 1) (Year 2 Score: 2) 


2 — Emerging 
Practice 


1 - Poor Practice 
with Action 


0 - Poor/No 
Practice 


Evaluation occurs 
somewhat 
iteratively and 
according to a 
somewhat 
transparent plan. 
Supervisors have 
some training on 
the rubric. 


Evaluation is not 
iterative or 
transparent. 
Supervisors have 
little or no training 
on the rubric. 


Poor practice with 
little to no action 
and/or lack of 
awareness of 
problems by top- 
level district 
leadership. 


Score change from last year. District leaders have taken strong steps in improving the 
principal evaluation process. Eighty-one percent of principals agree that the principal 
evaluation process is useful to principals (significantly higher than last year's 63%). Ninety- 
one percent of principals agree that the process is transparent and fair. They are both 
important improvements over time. Like last year, there is a well-defined performance 
evaluation process, and principal supervisors have worked to increase the number of 
meetings about the evaluation and to make them more authentic. Across all interviewees, 
there are almost all positive things to say. Principals, principal supervisors, and others 
understand and can describe the process consistently. All mention monthly meetings that are 
tied to the principal standards. While much of the district meetings (in fall 2020) were taken 
up with crisis planning related to COVID-19, there was still a strong effort to keep at least 
some of the monthly meeting time focused on improvement. Principal supervisors have done 
some norming so there is more reliability across the ratings; some admit more could be done 
here. As stated above, the more experienced and/or capable principals may not be getting 
as much out of the process as others, but they still find it useful. 


To improve this practice, more work needs to be done to keep up the momentum so that 
principals start to value the efforts that are being made, especially during the pandemic. In 
addition, more norming should be done to ensure that all principals are experiencing the 
same process. Finally, be sure that principal supervisors are able and willing to give direct, 
critical feedback for improvement, particularly to higher performing principals, where 
feedback may be more nuanced. 
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El Component 5: Execute, Reflect, and Improve 
Year 0 Score: N/A | Year 1 Score: N/A | Year 2 Score: 2 Year 3 Score: 3 


Model Practice 


Leaders avoid planning paralysis and instead execute high-quality implementation plans with urgency and 
asense of possibility. They have a bias to action, but they value time to pause, reflect, and adjust. 


Quick wins are achieved when possible to help build momentum. Teams achieve their goals on time - 
or adjust their goals with evidence when warranted. Teams have a sense of accountability and expect 
consequences if goals are unmet or changes are unexplained. 


Leaders have systems in place to measure progress by collecting data and checking in with those closest 
to the implementation. District leaders consistently use progress monitoring to define and analyze both 
bright spots and issue areas. Leaders identify and share learnings about what is working - or not - with 
stakeholders. 


District leaders use their reflection process to identify high-return improvements that make sense 
to implement in their context. They also stop doing what is not working. District leaders adjust 
their implementation plan when new information becomes available. Ongoing cycles of continuous 
improvement become deeply ingrained in the district. Stakeholders expect - and rely on - this 
methodology to guide their work. 


Current Practice and Recommendations 


Based on diagnostic data collected over the last year, we assigned District D an overall rating of 3 (Strong 
Practice) in the area of executing, reflecting, and improving. A majority of the district cohort team reports 
a sense of urgency when implementing initiatives, and, by and large, team members are motivated to 
take action. Departmental effectiveness surveys have been a powerful tool that the district has developed, 
allowing leaders to reflect and learn. The next step will be to take identified improvement goals and turn 
them into strategies to achieve these. Most stakeholders, principals, APs, and district leaders report that 
the district focuses on quick wins to build momentum. A large majority of the district cohort team reports 
achieving their goals, which builds on already strong perceptions over the past two years. In addition, 

the district cohort team adjusts the implementation plan as needed. Lastly, more district leaders and 
principals continue to perceive institutionalization of practices and policies year over year, and there are 
broad perceptions that the PTM work will continue to be prioritized and improved. 


To further strengthen this component, we recommend that the district do the following: 


1. Build on and continue to iterate the departmental feedback surveys. Tie the learnings and 
identification of improvement goals to improvement strategies. Ensure that this progress 
monitoring, reflection and learning, and improvement cycle is clearly and consistently 
communicated to stakeholders. 

2. Incorporate onboarding of new stakeholders, particularly district leaders who may be new to the 
district or new to the PTM work, into implementation planning. 
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The chart below illustrates the district's current performance across key aspects of continuously 
improving. 


Figure 6. Execute, Reflect, and Improve Indicator Scores by Year 


5a1. Take action 

5a2. Stay on track 

5a3. Generate quick wins 

5a4. Achieve goals 

5b1. Monitor progress 

5b2. Reflect and learn 

5c1. Improve strategies 

5c2. Adjust implementation plan 
5c3. Institutionalize Improvement 


Overall 


2.5 3 3.5 - 


@YearO @Year1 WYear2 MYear3 
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(Year 0 Score: N/A) 


4-Best 
Practice 


3 - Strong 
Practice 


2- Emerging 
Practice 


(Year 1 Score: N/A) 


1- Poor Practice with 
Action 


Below is an example of an indicator of El Component 5: Execute, Reflect, and Improve. 


5a. Achieve goals 


5a1.Take 
action 


(Year 2 Score: 2) 


0- Poor/No 
Practice 


District leaders 
sometimes balance 


District leaders District leaders 
avoid planning often avoid 
paralysis and planning planning and 
execute on high- Werle Gaia execution. They may 


quality WVelle=\aoceticmm not typically pause 


implementation on high-quality their work to reflect 
plans with implementation on actual versus 
urgency and a oEUKm@ iach mm Cesired progress or 
sense of action but may impact. 


possibility. They (Wyle pele. 
take action but when they feel 
pause when they Paice fel 

feel reflection, revision, or 
revision, or correction is/are 
correction is/are (qlte-XEi 
necessary. 


Relevant Evidence: 


with urgency. 


District leaders are often 
stuck in planning 
paralysis and rarely 
execute on high-quality 
implementation plans 
and reflect when 
necessary. 


Score change from last year. All of the district cohort team members agree that district 
leaders have a sense of urgency, as well as possibility, when implementing initiatives, which 
is higher than the district average of 84% and higher than the 83% reported a year ago. 
Furthermore, 89% of the district cohort team agree that district leaders effectively reflect, 
revise, and correct when implementing initiatives, which is above the district average of 83% 
and well above the 50% reported a year ago. It is exciting to witness these large year-over- 
year increases; moreover, the cohort team is increasingly relying on data to drive planning 
and implementation. The team is developing feedback surveys for each district department 
whereby the data is shared with district leaders, and "then they make action steps of what to 
improve, and they celebrate the good feedback that was received." 


To improve this practice, the district should continue to leverage its implementation plan 
while formalizing a reflection process for the data being collected from the department 
feedback surveys. Since the district cohort team is strong in its execution of the plan, one 
way to effectively prioritize reflection may be to incorporate reflection activities into the plan 
itself. Additionally, the district is at a critical phase to set bigger goals and take faster steps. 
The district should carefully utilize backward mapping to plan for and execute these goals 
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Poor practice with 
little to no action and/ 
or lack of awareness 
of problems by top- 
level district 
leadership. 
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